Website Markdown Crawler

Crawls a website starting from a seed URL and converts every page to clean Markdown optimized for LLM ingestion (LlamaIndex, LangChain, OpenAI, Pinecone). Output includes structured metadata per page: title, language detected, publication date, headings outline, word count, and chunked content ready for vector store upsert.

Features

• Clean Markdown Extraction: Strips noise (navigation, footers) to extract just the main content.
• Smart Chunking: Splits content into token chunks respecting paragraph boundaries.
• Language Filtering: Can automatically detect and filter pages by language (e.g., only en or fr).
• Domain Control: Keeps the crawler scoped to the seed URL's domain.
• Regex Exclusions: Skip non-valuable URLs like tags or author pages.

Output Example

Each crawled page yields a structured JSON record:

{
  "url": "https://docs.example.com/getting-started",
  "title": "Getting Started — Example Docs",
  "description": "Learn how to set up Example in 5 minutes.",
  "language": "en",
  "wordCount": 842,
  "tokenEstimate": 1120,
  "headings": [
    { "level": 1, "text": "Getting Started" },
    { "level": 2, "text": "Installation" }
  ],
  "markdown": "# Getting Started\n\nLearn how to...",
  "chunks": [
    { "index": 0, "content": "# Getting Started\n\nLearn how to...", "tokenEstimate": 498 }
  ],
  "chunkCount": 1,
  "depth": 1,
  "crawledAt": "2026-06-10T14:32:00.000Z"
}

Integrations

Connect the crawler directly into your RAG stack.

LlamaIndex

from llama_index.core import Document

# After running the Actor, download dataset as JSON
docs = [
    Document(text=chunk['content'], metadata={'url': item['url'], 'chunk': chunk['index']})
    for item in dataset_items
    for chunk in item['chunks']
]

LangChain

from langchain.docstore.document import Document as LCDoc

lc_docs = [
    LCDoc(page_content=chunk['content'], metadata={'source': item['url']})
    for item in dataset_items
    for chunk in item['chunks']
]